Minimization Problems Based on a Parametric Family of Relative Entropies I: Forward Projection

Minimization problems with respect to a one-parameter family of generalized relative entropies are studied. These relative entropies, which we term relative α-entropies (denoted Iα), arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the usual relative entropy (Kullback-Leibler divergence). Just like relative entropy, these relative α-entropies behave like squared Euclidean distance and satisfy the Pythagorean property. Minimizers of these relative α-entropies on closed and convex sets are shown to exist. Such minimizations generalize the maximum Rényi or Tsallis entropy principle. The minimizing probability distribution (termed forward Iα-projection) for a linear family is shown to have a power-law. Other results in connection with statistical inference, namely subspace transitivity and iterated projections, are also established. In a companion paper, a related minimization problem of interest in robust statistics that leads to a reverse Iα-projection is studied. Index Terms Best approximant; exponential family; information geometry; Kullback-Leibler divergence; linear family; power-law family; projection; Pythagorean property; relative entropy; Rényi entropy; Tsallis entropy.